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TOWARD  AN  OBSERVATION  SYSTEM 
FOR  MEASURING  LEADER  BEHAVIOR  IN  NATURAL  SETTINGS 


Abstract 


This  paper  reports  the  analysis  of  a  newly  developing  observation  system 
for  measuring  leader  behavior  in  natural  settings.  The  development  of  this 
leadership  observation  system  (LOS)  is  first  described  in  the  paper.  After 
trained  observers  had  logged  440  hours  of  free  observation  of  44  managers,  (10 
hours  each  over  a  two  week  period)  a  Delphi  approach  was  used  to  derive  12 
categories  and  accompanying  behavioral  descriptors.  Trained  participant  (N  * 
88)  and  outside  (N  =  8)  observers  simultaneously,  but  independently,  directly 
recorded  the  behavior  of  the  target  leaders  on  the  LOS  instrument  every  hour 
over  a  two  week  period.  The  target  leaders  also  filled  out  a  self  estimate  of 
time  usage  questionnaire  that  contained  the  same  12  categories  as  the  LOS 
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instrument.  In  addition,  the  target  leaders,  their  superiors  (N  **  118),  peers 
(N  =  210)  and  subordinates  (N  =  362)  completed  the  widely  used  LBDQ-XII  and 
the  new  MBS  (Yukl  &  Nemeroff,  1979)  questionnaires.  There  was  high 
interrater  agreement  reliability  between  the  participant  and  outside 
observers.  To  go  beyond  this  relatively  simple  reliability  assessment,  a 
multitrait-multimethod  (MTMM)  analysis  was  conducted.  The  results  gave  some 
support  for  the  validity  (both  convergent  and  discriminant)  of  the  LOS  when 
multiple  rater  sources  (participant  and  outside  observers)  were  treated  as 
more  than  one  method.  In  contrast,  neither  of  the  leadership  questionnaire 
measures  (LBDQ-XII  or  MBS)  was  demonstrated  to  have  any  support  for  construct 
validity  when  multiple  rater  sources  (self,  superior,  peers  and  subordinates) 
were  treated  as  multiple  methods.  When  the  standardized  questionnaires  and 
leadership  observation  system  were  treated  as  multiple  methods,  the  validity 


analysis  was  not  very  encouraging.  Part  of  the  problem,  however,  was  that 

there  were  not  directly  comparable  behavioral  categories  across  these  .  " 

methods.  When  directly  comparable  categories  from  the  seif  estimate  of  time  [jj 

usage  questionnaire  were  compared  to  the  LOS,  the  MTMM  analysis  yielded  more  C.i 
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support  for  validity.  Although  this  study  is  clearly  only  a  beginning,  there  .i _ 

seems  to  be  enough  initial  support  for  reliability  and  validity  to  warrant 
further  development  and  analysis  of  this  observational  system  for  measuring  y 
leader  behavior  in  natural  settings.  Availability  Co4es 
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Toward  An  Observation  System  for  Measuring 
Leader  Behavior  in  Natural  Settings 

At  the  Fifth  Biennial  Leadership  Symposium  a  social  learning  theory  base 
and  observational  measurement  of  leader  behavior  were  proposed  (Luthans, 

1979).  More  specifically  a  call  was  made  for:  (1)  getting  back  to  observable 
behavior  in  natural  settings  as  the  unit  of  analysis  for  the  study  of  leader¬ 
ship;  (2)  a  social  learning  theoretical  framework  that  recognized  leadership 
to  be  a  reciprocal,  interactive  process  involving  the  leader  (including 
his/her  cognitions  and  traits),  the  environment  (including  followers  and 
structural  and  other  organizational  and  broader  environmental  variables),  and 
the  leader's  behavior  itself;  and  (3)  alternatives  to  the  commonly  used  in¬ 
direct  questionnaire  measures  of  leader  behavior  such  as  an  observational 
system. 

A  number  of  papers  and  book  sections  have  since  attempted  to  refine  and 
expand  the  first  two  points  either  directly  (e.g.  see:  Davis  &  Luthans,  1979; 
Luthans,  1981,  pp.  429-432;  Luthans  &  Davis,  1979)  or  indirectly  (e.g.  see: 
Davis  &  Luthans,  1980a,  1980b;  Luthans,  1981,  pp.  63-71;  Luthans  &  Davis, 

1982;  Luthans,  Paul  &  Baker,  1981).  The  interested  reader  is  referred  to  these 
sources  for  a  full  treatment  of  the  use  of  observable  behavior  as  the  unit  of 
analysis  and  social  learning  as  the  theoretical  base  for  the  study, 
understanding,  and  research  perspective  for  leadership. 

Let  it  simply  be  said  that  the  study  described  here  drew  from  the  first 
proposal  of  the  earlier  symposium  by  using  observable  behavior  in  situ  as  the 
unit  of  analysis  and  from  the  second  proposal  by  using  the  interactive  notion 
from  social  learning  as  the  theoretical  foundation.  The  major  thrust  of  this 
paper,  however,  is  to  explore  and  report  the  efforts  made  on  the  third  propos¬ 
al  made  at  the  earlier  symposium  -  the  need  to  develop  other  than 
questionnaire  measures  of  leader  behavior  such  as  an  observation  system. 
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Why  an  Observation  System  to  Measure  Leader  Behavior 

An  observational  measurement  approach  to  leadership  behavior  seems  desir¬ 
able  for  two  major  reasons.  First,  if  leadership  is  viewed  as  an  interaction¬ 
al  process,  as  in  social  learning  theory,  then  as  Kerlinger  (1973)  has  pointed 
out,  "observations  must  be  used  when  the  variables  of  research  studies  are 
interactive  and  interpersonal  in  nature"  (p.  554).  The  second  reason  for  the 
need  of  an  observational  system  is  the  apparent  inadequacies  of  existing 
questionnaire  measures.  There  is  a  growing  awareness  and  recognition  that  the 
questionnaire  measures  that  leadership  research  has  almost  solely  depended 
upon  over  the  years  may  be  a  major  reason  for  the  dismal  state  of  the  field. 

Schriesheim  and  his  colleagues  have  recently  supplied  empirical  evidence 
that  casts  some  serious  doubts  as  to  the  reliability  and  validity  of  commonly 
used  leadership  questionnaires  such  as  the  Ohio  State  LBDQ  and  Fiedler’s  LPC 
instruments  (e.g.,  see:  Schriesheim,  Bannister  &  Money,  1979;  Schriesheim  & 
Kerr,  1974,  1977;  Schriesheim,  Kinnicki  &  Schriesheim,  1979).  For  example,  in 
a  comprehensive  paper  presented  at  the  Fourth  Biennial  Leadership  Symposium, 
Schriesheim  and  Kerr  (1977)  concluded  that  "leadership  is  today  without  any 
instruments  of  demonstrated  validity  and  reliability"  (p.  33).  This  finding 
does  not  necessarily  mean  that  the  instruments  reviewed  do  not  possess  reli¬ 
ability  and  validity,  but  rather  the  available  evidence  simply  does  not  sup¬ 
port  them.  This  has  spurred  a  number  of  studies  to  demonstrate  the  psychome¬ 
tric  properties  of  these  instruments  (see  Bass,  1981  for  a  review  of  this 
literature)  and  the  more  careful  development  of  new  questionnaires  (Yukl  & 
Nemeroff,  1979).  However,  except  for  a  few  preliminary  attempts  (for  example 
the  last  Symposium  had  two  papers  that  were  based  on  observational  research, 
see:  Bussom,  Larson  &  Vicars,  1982;  Lombardo  &  McCall,  1982;  and  this  current 
Symposium  contains  an  observational  study  of  educational  managers  reported  by 
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Martinko  and  Gardner,  1982),  more  direct  alternative  measurement  techniques 
such  as  observational  systems  have  not  been  developed  nor  used  In  leadership 
research.  The  purpose  of  this  paper  is  to  provide  at  least  the  beginnings  of 
a  possible  alternative  -  a  supplement  (not  a  total  replacement)  to  question¬ 
naire  measures  of  leader  behavior.  This  study  was  undertaken  to  make  a 
preliminary  assessment  of  a  newly  developing  leader  observation  system  (LOS) 
for  the  measurement  of  leader  behavior  in  natural  settings. 

Method 


Settings  and  Subjects 

The  study  utilized  five  purposely  diverse  organizational  samples:  a 
fairly  large  financial  instituition,  a  state  agency,  a  medium  sized 
manufacturing  plant,  a  campus  police  department,  and  the  Navy  and  Army 
R.O.T.C.  units  of  a  university.  All  those  with  supervisory  responsibilities 
(from  the  presiv ^nt  down  to  first  line  supervision)  in  the  financial  institui¬ 
tion  (N=52),  campus  police  (N=16),  and  professional  staff  in  the  R.O.T.C. 
units  (N=15)  and  supervisors /managers  (usually  within  the  n^me  department)  in 
selected  operational  departments  of  the  state  agency  (N=18)  and  manufacturing 
plant  (N“19)  served  as  target  leaders  (total  N-120)  in  the  study.  Thus, 
leaders  are  defined  in  this  study  as  those  in  managerial  positions  (at  all 
levels)  with  responsibilities  for  supervising  two  or  more  subordinates.  These 
target  leaders  typically  had  been  with  their  respective  organizations  6-10 
years  and  in  their  present  positions  1-5  years.  They  were  generally 
distributed  throughout  the  26-55  age  range  and  a  great  majority  had  a  college 
education.  Their  jobs  covered  the  whole  range  of  functions  found  in  their 
respective  organizations.  These  demographics  were  compatible  with  the  intent 
of  the  study  which  was  to  generalize  across  levels,  functions  and  personal 
characteristics  of  leaders. 
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Measures 

Leadership  behavior  measures  included  In  this  study  were  the  Leader 
Behavior  Description  Questionnaire — Form  XII  (LBDQ-XII)  developed  by  Stogdill, 
Goode,  and  Day  (1962),  the  Managerial  Behavior  Survey  (MBS)  developed  by  Yukl 
and  Nemeroff  (1979),  and  the  newly  developing  Leadership  Observation  System 
(LOS). 

The  LBDQ-XII  was  used  because  it  represents  the  most  widely  used  measure 
in  leadership  behavior  research  to  date.  Many  of  the  leadership  theories  and 
research  findings  to  date  are  based  on  some  variation  of  the  IS  (initiating 
structure)  and  C  (consideration)  subscales  included  in  this  questionnaire. 

Yet,  as  previously  pointed  out,  the  LBDQ-XII  measure  lacks  convincing  demon¬ 
strable  support  for  construct  validity.  Previously  reported  reliabilities  of 
the  LBDQ-XII  have  been  fairly  favorable  (Bass,  1981;  Schriesheim  &  Kerr,  1977) 
and  the  Cronbach  alphas  in  the  present  study  (N=393)  ranged  from  .74  to  .91 
for  the  various  subscales «  The  IS  scale  was  .86  and  the  C  scale  was  .83. 

Although  a  relatively  new  and  still  developing  instrument,  the  MBS  was 
used  because,  in  the  words  of  the  authors  Yukl  and  Nemeroff  (1979),  it  was 
specifically  designed  to  "identify  distinct,  meaningful,  and  widely  applicable 
categories  of  leadership  behavior"  (p.  169).  Thus,  it  attempts  to  tap  multi¬ 
ple  behaviors.  In  addition,  the  reported  psychometric  properties  of  the  MBS 
reported  by  its  authors  is  quite  favorable.  In  four  separate  studies  the 
Cronbach  alphas  are  reported  in  the  .7  and  .8  range  (Yukl  &  Nemeroff,  1979) 
and  in  the  present  study  (N«395)  they  ranged  from  .56  to  .90  on  the  various 
subscales. 

The  Leader  Observation  System  (LOS) 

The  Observation  System  used  in  this  study  was  developed  in  two  major 
phases.  First,  44  leaders  (defined  as  those  in  managerial  positions  with 
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supervisory  responsibilities)  at  all  levels  in  all  types  of  organizations  (not 
the  120  target  leaders  who  were  the  subjects  of  the  actual  study)  were 
observed  in  a  completely  unstructured  forma*:  for  a  varied  hour  each  day  over  a 
two  week  period  of  time  (i.e.  440  hours  of  unstructured  observation  of  leaders 
in  their  natural  settings).  The  44  observers  were  management  students  who 
were  given  an  extensive  training  workshop.  This  observer  training  emphasized 
the  systematic  errors  commonly  found  in  observing  others  (i.e.  it  followed  the 
procedures  suggested  by  Thornton  &  Zorich,  1980).  In  addition,  they  practiced 
writing  protocols  from  several  role  playing  exercises  that  were  then  critiqued 
by  the  trainers/researchers.  The  observers  were  trained  to  continuously 
observe  the  behavior  of  the  target  leader  over  the  hour;  to  record  specific, 
identifiable  behaviors  on  their  logs;  and  to  be  reporter'-  concentrating  on 
objective  description  rather  than  trying  to  judge  or  evaluate  the  behaviors 
observed.  These  observers  had  not  yet  studied  leadership  theory  or  research, 
thus  the  attempt  was  made  to  minimize  the  possible  confounding  effect  that 
implicit  theories  may  have  on  observing  leader  behavior  (DeNisi  &  Schriesheim, 
1981). 

While  true  randomization  of  the  observation  times  was  not  possible,  the 
observers  systematically  varied  their  hours  throughout  each  working  day  during 
the  two  weeks  to  help  assure  representativeness.  After  the  two  weeks,  the 
observed  leaders  were  shown  copies  of  the  protocals  of  their  behaviors  and 
were  asked  to  rate  to  what  extent  these  were  typical  of  their  behavior.  On  a 
scale  of  1-5,  the  mean  rating  was  3.9,  which  indicated  the  behaviors,  on  the 
average,  were  typical  "to  a  considerable  extent."  These  leaders  were  also 
asked  to  suggest  any  additional  behaviors  which  they  considered  typical. 

These  additions  mainly  consisted  of  activities  which  might  be  best  described 
as  of  a  sensitive  nature,  i.e.,  important  policy  meetings,  disciplining,  and 
managing  conflict. 
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The  second  major  phase  involved  in  deriving  the  LOS  used  in  this  study 
was  the  considerable  job  of  constructing  comprehensive  and  workable  categories 
to  accommodate  (contain)  the  440  hours  of  freely  observed  behaviors.  This 
task  was  accomplished  by  a  Delphi  process  (Delbecq,  VandeVen,  &  Gustafson, 
1975).  The  Delphi  panel  consisted  of  four  persons  with  considerable  academic 
work  in  management /leadership  and  three  graduate  students  from  outside  disci¬ 
plines  who  were  completely  naive  with  respect  to  prior  leadership  research. 

All  panel  members  were  given  handouts  and  required  to  read  and  become  familiar 
with  the  processes  of  constructing  adequate  behavioral  categories  as  outlined 
by  Kerlinger  (1973)  and  Crano  and  Brewer  (1973). 

In  the  first  Delphi  round,  the  panelists  independently  reviewed  the 
extensive  protocals  completed  by  the  observers  and  suggested  general  categor¬ 
ies  with  accompanying  behavioral  descriptors  from  the  protocals.  The  panel¬ 
ists  were  instructed  to  use  frequency  of  behaviors  as  a  guideline  in 
constructing  the  categories.  These  categories  with  accompanying  comments  were 
collected  (there  were  about  100  categories  resulting  from  the  first  round)  and 
fed  back  to  the  panelists.  Then  through  several  iterations  the  panelists 
further  collapsed  the  categories  into  smaller  but  more  comprehensive  sets 
which  could  be  readily  used  by  observers  to  record  the  frequency  of 
occurance.  Through  this  Delphi  process,  the  final  surviving  12  categories 
incorporated  a  multiplicity  of  opinions  and  critiques  whose  purpose  was  not 
only  to  be  representative  but  also  to  be  as  exhaustive  and  mutually  exclusive 
as  possible.  The  resulting  12  final  categories  and  the  accompanying  behavior¬ 
al  descriptors  on  the  LOS  instrument  are  shown  in  Figure  1. 

Insert  Figure  1  About  Here 
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The  categories  in  the  LOS  are  conceptually  similar  to  those  associated 
with  a  managerial  activities  approach.  Both  the  LOS  and  a  managerial  activi¬ 
ties  approach  attempt  to  determine  what  managers/leaders  actually  do  in  the 
natural  setting  and  use  direct  methods  of  measurement.  However,  the 
traditional  behavioral  approaches  to  leadership  (e.g.  the  Ohio  State  or 
Michigan  studies)  depend  on  indirect  questionnaire  measures  and  the  categories 
for  these  measures  were  not  derived  from  free  observation  of  leaders  in 
natural  settings.  Instead,  the  researchers  themselves  largely  determined  the 
response  sets  of  their  questionnaires  and  the  resulting  behavioral  categories 
are  quite  different  (i.e.  usually  less  concrete)  from  the  LOS  categories. 

Yukl  and  Nemeroff's  MBS  instrument,  on  the  other  hand,  does  contain  many 
similar  categories.  Although  drawn  from  the  literature  rather  than  free 
observation  in  natural  settings,  they  did  make  a  conscious  effort  to  select 
observable,  concrete  behavioral  categories  (Yukl  A  Nemeroff,  1979). 

Mintzberg  (1973,  1975)  is  most  closely  associated  with  the  managerial 
activities  approach,  but  the  "leader"  is  only  one  of  his  ten  managerial 
roles.  He  states  that,  "leadership  involves  interpersonal  relationships 
between  the  leader  and  the  led,"  (1973,  p.  60).  As  such,  the  manager  must 
engage  in  activities  which  provide  guidance  to  subordinates,  motivate  them, 
and  create  favorable  conditions  for  the  work.  Some  of  these  managerial  activ¬ 
ities  may  be  classified  as  primarily  concerned  with  leadership  (an  example 
would  be  staffing  which  involves  hiring,  training,  evaluating,  renumerating, 
promoting,  and  dismissing  subordinates).  Mintzberg,  however,  makes  it  clear 
that  leadership  permeates  all  managerial  activities,  even  those  with  some 
other  basic  purpose.  When  a  manager  requests  information  from  a  subordinate, 
for  example,  he  or  she  may  be  simultaneously  motivating,  training,  allowing 
participation  in  decision-making,  and/or  monitoring  the  subordinate's  perform- 
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ance.  Consequently,  it  appears  to  be  difficult  to  separate  "leadership  behav¬ 
ior",  per  se,  from  the  larger  domain  of  managerial  activities  and  the  LOS 
reflects  this  difficulty. 

Results  from  Mintzberg-type  studies  (e.g.  see:  Kurke  &  Aldrich,  1979) 
using  different  organizations  and  subjects  tend  to  confirm  his  findings.  In 
general,  these  studies  suggest  that  subordintes  consume  about  one-half  of 
managers'  contact  time  and  the  purpose  of  these  contacts  usually  involved 
requests,  sending  or  receiving  information,  and  occasionally  strategy¬ 
making.  These  studies,  however,  report  encountering  some  difficulties  in 
coding  "purpose  of  activities"  largely  because  of  the  ambiguous  or  overlapping 
nature  of  the  categories.  In  other  words,  the  coding  of  activity  purpose 
often  requires  a  great  deal  of  inference  on  the  part  of  the  observer  to 
discriminate  between,  for  example,  overt  and  covert  purposes,  sequential 
purposes,  multi-purposes,  and  changed  purposes  (Mintzberg,  1973,  pp.  274— 

276).  The  LOS  tries  to  overcome  this  coding  problem  by  only  dealing  with 
frequences  of  observable  behavior  and  does  not  require  inference  on  the  part 
of  the  observer. 

The  actual  format  of  the  LOS  instrument  used  in  the  present  study  lists 
the  behavioral  categories  along  the  left  hand  side  and  random  times  along  the 
top.  The  random  times  were  for  10  minutes  every  hour  over  two  weeks  or  a 
total  of  80  observations.  There  was  a  separate  sheet  for  each  day.  A  nominal 
measuring  format  was  used;  i.e.  the  observers  recorded  either  the  behavior  was 
present  (”  /  ")  or  absent  ("0")  for  each  10  minute  time  slot.  By  judging 

whether  the  behavior  was  present  or  absent  the  problem  of  inferring  covert 
purposes  of  the  behavior  or  degrees  or  magnitude  of  the  behavior  was 
avoided.  Only  a  frequency  count  of  the  behavior  was  recorded  by  the  LOS  In 


this  study 


Observers  Used  in  the  Study 

I’articipant  observers  (N“88)  in  the  study  were  selected  jointly  by  the 
researchers  and  the  personnel  managers  of  the  respective  organizations  (or  the 
designated  project  officers  in  the  case  of  the  campus  police  and  R.O.T.C. 
units)  according  to  the  following  criterion:  Does  this  person  have  maximum 
visual  and  audible  contact  with  the  target  leader  and  have  a  good  understand¬ 
ing  of  the  functions,  terminology,  and  nature  of  the  work  performed  by  the 
target  leader.  The  target  leader's  informed  consent  was  also  needed  and 
secured  in  all  cases. 

The  selected  participant  observers  in  almost  all  cases  turned  out  to  be 
the  target  leader's  secretary  or  a  key  subordinate.  Eleven  (12%)  of  the 
participant  observers  were  responsible  for  observing  two  target  leaders  and  7 
(8%)  of  the  participant  observers  had  three  target  leaders.  This  was  discour¬ 
aged  as  much  as  possible  and  only  occurred  when  in  the  opinion  of  the 

researchers/personnel  managers  it  was  better  to  meet  the  criterion  of  selec¬ 
tion  as  a  participant  observer  in  the  study  and  observe  more  than  one  target 

leader  than  select  another  observer  but  not  meet  the  selection  criterion 
nearly  as  well.  This  usually  was  the  case  where  one  secretary  served  more 
than  one  target  leader.  These  participant  observers  had  considerable  job 
experience  but  little  formal  higher  education  and  had  little  or  no  knowledge 
of  the  literature  on  leadership  research  or  theory,  thus  minimizing  the  impli¬ 
cit  theory  problem  (DeNisi  &  Schrlesheim,  1981).  Except  for  the  training  they 
received  (which  will  be  described  next),  they  had  little  or  no  knowledge  of 
the  specifics  of  the  study. 

The  outside  observers  (N=8)  used  in  the  study  were  graduate  students  in 
management.  Three  were  assigned  to  the  financial  institution,  two  to  the 
manufacturing  plant,  and  one  to  the  state  agency  and  one  to  the  campus  police 


and  R.O.T.C.  units.  An  experienced  graduate  student  (a  Ph.D.  student  with  the 
most  knowledge  of  the  study)  observed  at  all  the  sites  and  largely  coordinated 
the  efforts  of  the  other  7  outside  observers.  These  outsider  observers  had 
briefly  studied  leadership  theory  and  research  in  their  course  work  in  manage¬ 
ment,  received  training  which  will  be  described  next  and  had  a  very  general 
idea  of  the  objectives  and  procedures  of  the  study. 

Observer  Training 

A  training  workshop  conducted  by  the  researchers  was  held  on  the  premises 
of  each  of  the  participant  observers'  respective  organizations.  Each  session 
followed  the  same  format,  used  the  same  trainers  (the  authors  of  this  paper 
plus  a  graduate  assistant)  and  took  approximately  2  1/2  hours  to  complete. 

About  the  first  half  of  the  observer  training  workshop  was  devoted  to 
three  areas:  First,  to  provide  a  very  general  explanation  of  the  purpose  of 
the  observations  (i.e.  to  gather  data  for  input  into  a  profile  of  the  leader's 
behavior);  second,  to  go  over  in  detail  the  observational  instrument,  giving 
special  attention  and  analysis  to  the  12  behavioral  categories  and  the  pro¬ 
cedures  for  filling  out  the  instrument  including  what  to  do  if  the  leader  was 
absent;  and  third,  to  give  careful  instruction  on  potential  observational 
errors  (following  Thornton  &  Zorich,  1980)  and  how  to  overcome  them.  In 
particular,  the  potential  errors  of  description  versus  evaluation  and  distor¬ 
tion  to  please  the  person  being  observed  were  deemed  to  be  particularly  rele¬ 
vant  to  these  participant  observers  and  were  stressed  in  the  training.  For 
example,  the  observers  were  instructed  to  avoid  letting  their  evaluative 
biases  color  their  observations,  since  there  are  no  "good"  or  "bad"  categories 
on  the  instrument,  the  observations  would  be  useful  only  if  they  were  accu¬ 
rate.  By  careful  explanation  and  example  the  trainers  showed  how  the  observ¬ 
ers  could  avoid  these  errors. 
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The  second  half  of  the  training  was  devoted  to  demonstration  and  prac¬ 
tice.  The  trainers  employed  a  number  of  role-playing  skits  which  illustrated 
the  specific  leader  behavior  categories,  and  the  trainees  used  the  instrument 
to  record  the  behaviors  they  observed.  By  following  the  principles  of  model¬ 
ing  theory  (Bandura,  1977;  Latham  &  Sari,  1979)  this  aspect  of  the  training 
was  intended  to  increase  observer  accuracy  through  modeling,  rehearsal,  and 
repetition.  After  each  role-playing  skit,  the  trainers  went  over  the  LOS 
instrument  with  the  trainees  and  discussed  which  behavior  category  was  being 
illustrated  and  which  specific  errors  might  have  been  committed  during  that 
observation. 

In  a  final  role-playing  skit,  which  was  rather  lengthy  and  elaborate  but 
realistic,  6  behavioral  categories  were  represented.  The  observers  perform¬ 
ance  on  this  last  exercise  served  as  an  evaluation  check  for  the  training.  A 
precise  evaluation  of  observer  accuracy  is  possible,  of  course,  only  when 
there  is  an  objective  criterion,  i.e,  when  the  "correct"  observations  are 
known.  Such  an  objective  criterion  was  possible  in  this  training  exercise 
because  the  skit  was  designed  to  exhibit  the  6  categories  and  thus  an  evalua¬ 
tion  of  trainee  accuracy  could  be  made.  Although  this  data  were  unavailable 
in  one  of  the  organizations,  in  the  remaining  four  organizations  the  partici¬ 
pant  observer  trainees  had  an  overall  mean  accuracy  of  92.5%,  with  no  signifi¬ 
cant  differences  between  organizations.  This  accuracy  was  considerably  higher 
than  the  69%  obtained  by  Thornton  and  Zorich  (1980)  in  their  observer  training 
group,  but  there  exercise  was  longer. 

The  outside  observers  used  in  the  study  were  given  the  same  training  as 
the  inside,  participant  observers.  After  the  training,  they  were  given  a  tour 
of  the  facility  and  were  introduced  and  chatted  with  the  participant  observers 
they  would  be  working  with  over  the  two  week  observation  period. 
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Data  Collection  Procedures 

Approximately  four  weeks  prior  to  the  collection  of  the  LOS  data,  all 
target  managers  (N»120)  completed,  among  other  instruments,  the  MBS  and  LBDQ- 
XII.  The  wording  on  these  instruments  were  changed  to  reflect  a  self  assess¬ 
ment.  They  were  completed  on  site  in  the  respective  organizations  under  the 
supervision  and  instructions  of  the  researchers.  The  four  week  time  lag 
between  the  questionnaire  data  collection  and  the  observational  data  collec¬ 
tion  was  used  in  order  to  minimize  contamination  effects.  The  target  leaders 
were  also  asked  to  fill  out  a  brief  questionnaire  that  asked  them  to  give  a 
self-estimate  of  the  percentage  of  their  work  time  spent  in  each  of  the  LOS 
categories. 

Each  target  leader  then  distributed  the  LBDQ-XII  and  MBS  questionnaires 
to  their  immediate  superior  (N=*118),  usually  two  peers  (N=210),  and  about 
three  subordinates  (N=362).  The  pro-nouns  and  instructions  were  modified  in 
each  of  these  samples  to  reflect  who  was  filling  it  out.  If  a  target  manager 
directly  supervised  a  large  number  of  subordinates  the  researchers  randomly 
selected  out  a  sample  of  three  to  five  subordinates  to  fill  out  the  question¬ 
naires.  It  was  stressed  the  anonymity  of  these  raters  would  be  preserved  and 
their  names  never  appeared  with  the  data. 

The  LOS  data  was  then  collected  on  the  target  leaders  80  times  by  the 
participant  observer  over  a  two  week  period  of  time  (a  random  10  minute  period 
each  working  hour  over  two  weeks).  This  represented  a  total  of  9600  (120 
target  managers  X  80  observation  periods)  possible  observation  periods  when 
the  LOS  instrument  was  filled  out.  In  addition,  all  participant  observers 
completed  a  short  follow-up  questionnaire  at  the  conclusion  of  their  two-week 
observational  period.  The  purpose  of  this  questionnaire  was  to  assess  the 
extent  to  which  observed  behaviors  were  representative  or  "typical"  of  normal 
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behavior  patterns  exhibited  by  the  target  leaders  over  the  time  they  have 
worked  with  them.  That  is,  the  leaders  may  have  attempted  to  act  in  exemplary 
ways  not  necessarily  customary  to  their  typical  behavior  patterns  simply 
because  of  the  knowledge  that  they  were  being  observed  or  this  two  week  period 
may  not  have  been  representative  of  their  typical  work  load.  Results  for  this 
Representativeness  Scale  indicated  that  all  observed  behavior  categories  on 
the  LOS,  on  average,  were  reported  to  be  typical  to  a  considerable  or  very 
great  extent. 

Since  it  was  not  feasible  to  have  the  outside  observers  present  at  all 
times  during  the  observation  data  collection  periods,  a  time  sampling  tech¬ 
nique  was  employed  to  gather  their  data.  The  trained  outside  observer  would 
randomly  appear  unannounced  and  simultaneously  with  the  participant  observers, 
but  independently,  record  on  the  LOS  sheets  the  observed  behaviors  of  the 
target  leader.  A  total  of  253  such  simultaneous,  independent  observations 
took  place.  Each  of  the  participant  observers  had  two  or  three  times  when  an 
outside  observer  joined  them  unannounced  to  simultaneously  record  the  behavior 
of  the  target  leader. 

A  summary  of  procedures,  in  order  of  sequence,  is  the  following: 

1.  LBDQ-XII  and  MBS  questionnaire  administration  to  target 

leaders  (N**120)  and  other  rater  sources  (superior 
(N«118),  peers  (N"*120,  and  subordinates  (N»362))  and 
target  leader  questionnaire  on  self  estimate  of  time 
use 

2.  Observer  training 

3.  Measurement  of  target  leaders’  behavior  using  the  LOS 

instrument  filled  out  by  participant  observers  every 
10  minutes  for  two  weeks  and  periodically  by  outside 
observers 
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Reliability  and  Validity 

The  development  of  any  new  measurement  system  must  address  the  important 
psychometric  issues  of  reliability  and  validity.  Unless  the  measurement 
scheme  can  be  demonsrated  to  be  dependable,  consistent  and  accurate,  i.e. 
reliable,  there  is  always  the  possibility  that  the  data  gathered  by  the 
instrument  are  loaded  with  error  and  the  results  are  meaningless.  To  the 
extent  which  measurement  error  is  demonstrated  to  be  slight  or  minimal,  the 
measure  is  said  to  be  reliable  (Nunnally,  1978).  The  most  common  way  to 
assess  the  reliability  of  behavioral  data  gathered  by  observation  has  been 
through  interrater  agreement  (Bijou,  Peterson  &  Ault,  1968). 

Interrater  reliability  analysis  for  the  LOS  is  quite  encouraging  (see 
Luthans,  Lockwood  &  Conti,  1981  for  a  complete  discussion  and  a  full  data 
presentation  of  the  interrater  reliability  assessment).  There  is  93.5  percent 
agreement  between  the  participant  observers  and  the  outside  observers  in  this 
study.  When  only  agreement  on  observed  behaviors  (leaving  out  agreement  on 
behavioral  categories  that  did  not  occur)  is  calculated,  there  is  87.4  inter¬ 
rater  reliability.  Although  reliabilities  are  not  even  reported  in  but  a 
couple  of  observational  studies  involving  managerial  leadership  (Bussom 
et.al.,  1981),  in  order  to  rule  out  the  possibility  of  chance  agreement, 
statistical  analysis  was  also  performed  on  the  interrater  data  gathered  by  the 
LOS  instrument.  Chi  square  calculations  yielded  highly  significant  values  (p 
<  .001)  ranging  from  =*  99.3  for  "staffing"  to  =  119.3  for 
"Monitoring/Controlling  Performance"  and  the  r^  statistics  (Cohen  &  Cohen, 
1975)  had  highly  significant  values  (p  <  .001)  ranging  from  .89  for  "Staffing" 
to  .68  for  "Monitoring/Controlling  Performance."  Perhaps  the  most  revealing 
statistic,  however,  is  Cohen's  (1960)  kappa  statistic  which  specifically 
represents  the  proportion  of  joint  judgments  in  which  there  is  agreement. 
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after  chance  agreement  is  excluded.  The  values  for  this  kappa  statistic  are 
very  similar  to  those  for  r^  and  the  overall  was  a  highly  significant  (p  < 
.001)  .81. 

The  interrater  agreement  assessment  makes  a  contribution  to  the  reliabil¬ 
ity  of  the  LOS.  However,  more  and  different  analysis  is  also  needed.  In 
addition,  more  important  validity  analysis  is  needed.  Reliability,  of  course, 
is  a  necessary  but  not  sufficient  condition  for  validity.  A  measurement 
scheme  may  be  consistently  and  accurately  measuring  something  other  than  the 
intended  construct.  Validity  basically  refers  to  whether  the  measurement 
system  is  measuring  what  it  is  supposed  to  measure  (in  this  case  does  the  LOS 
really  measure  leader  behavior).  This  simple  definition  does  not  intend  to 
imply  that  validity  can  be  demonstrated  by  a  single  study.  Compared  to  reli¬ 
ability,  validity  is  much  more  difficult  to  demonstrate. 

This  study  uses  the  multitrait-multimethod  (or  simply  MTMM)  approach  to 
extend  the  relatively  simple  interrater  agreement  assessment  of  reliability 
and  helps  begin  the  evaluation  of  validity.  Cascio  points  out  that  the  MTMM 
analysis  contributes  to  both  reliability  and  validity  assessment.  He  states: 
"reliability  is  estimated  by  two  measures  of  the  same  trait  using 
the  same  method,  while  validity  is  defined  as  the  extent  of 
agreement  between  two  measures  of  the  same  trait  using  different 
methods.  Thus,  the  distinction  between  reliability  and  validity 
is  simply  a  matter  of  degree — that  is,  in  terms  of  the  similarity 
of  measurement  methods"  (1978,  p.  97). 

Besides  helping  to  assess  both  reliability  and  validity,  the  MTMM  focused 
on  construct  rather  than  predictive  validity.  The  evaluation  of  construct 
validity  seems  relatively  more  important  at  this  early  stage  of  development  of 
the  LOS.  As  Guion  points  out,  "All  validity  is  at  its  base  some  form  of 


construct  validity.  .  .  The  most  salient  of  the  traditionally  identified 
aspects  of  validity — the  only  one  that  is  salient — is  construct  validity.  It 
is  the  basic  meaning  of  validity"  (1977,  p.  410).  Eventually,  if  subsequent 
studies  can  build  support  for  construct  validity,  then  predictive  validity 
will  become  more  important.  Once  again,  however,  it  must  be  remembered  that 
this  study  is  only  a  beginning.  Reliability,  and,  to  a  greater  extent  valid¬ 
ity  assessment,  is  both  a  logical  and  an  empirical  process.  The  goal  over 
time  is  to  build  a  type  of  nomological  network  to  assess  the  LOS  and  leader¬ 
ship  in  general.  In  this  network,  observable  leader  behaviors  would  be  re¬ 
lated  to  other  observables,  observables  to  theoretical  constructs,  and  one 
theoretical  construct  to  another  theoretical  construct.  Only  a  portion  of 
this  long  term  goal  is  realized  in  this  present  study. 

Results  of  the  MTMM  Analysis 

The  MTMM  analysis  in  this  study  mainly  depended  on  multiple  rater  sources 
as  measurement  by  more  than  one  method.  Some  may  argue  that  such  a  multirater 
comparison  makes  more  of  a  reliability  than  validity  assessment.  As  used  in 
this  study,  this  approach  hopefully  contributes  to  both  reliability  and  valid¬ 
ity  assessment.  There  are  important  precedents  where  multiple  raters  were 
used  in  analyzing  construct  validity.  For  example,  as  Lawler  (1967)  carefully 
noted: 

Campbell  and  Fiske  (1959)  consider  the  multitrait-multimethod 
approach  rather  than  the  multitrait-multirater  approach;  however, 
they  point  out  that  use  of  raters  that  occupy  different  organiza¬ 
tional  positions  relative  to  the  ratee  can  reasonably  be 
considered  to  be  measurement  by  more  than  one  method  (p.  372). 


In  addition,  it  is  important  to  point  out  that  one  of  the  examples  used  in  the 


original  Campbell  and  Kiske  (1959,  p.  96)  paper  uses  multiraters  in  the  MTMM 
matrix.  Thus,  multiraters  are  first  treated  as  multimethods  in  the  MTMM 
analyses  made  of  the  LBDQ,  MBS  and  LOS.  This  is  then  followed  by  an  MTMM 
analysis  that  uses  observation  and  questionnaire  methods  as  multimethods.  The 
problem  with  the  latter  approach,  however,  is  that  there  are  not  directly 
comparable  categories  of  behavior  across  the  methods. 

MTMM  Analysis  of  the  LBDQ-XII 

A  correlation  matrix  for  the  LBDQ-XII  was  constructed.  Correlation 
coefficients  between  different  rater  sources  were  calculated  using  the  Pearson 
Product  Moment  formula.  In  general,  the  matrix  revealed  that  different  rater 
sources  (subordinate,  superior,  peer,  and  self)  did  not  tend  to  agree  highly 
enough  (on  the  validity  diagonals)  to  argue  that  convergent  validity  existed 
to  any  great  extent.  One  possible  exception  was  between  superior  and  self 
ratings  where  six  of  the  twelve  correlations  were  slightly  significant  in  the 
•20's  range,  but  not  enough  to  provide  strong  support  for  convergent  validity 
between  these  two  rater  sources.  In  addition,  the  results  did  not  support 
discriminant  validity  among  the  behavioral  categories  on  the  LBDQ-XII. 

Kavanaugh,  MacKinney  and  Wolins  (1971)  provide  a  simplified  and 
interpretable  technique  for  quantitatively  analyzing  and  summarizing  large 
MTMM  matrices.  In  addition,  this  statistical  analysis  is  less  subject  to 
judgemental  interpretation.  In  essence,  theii  analysis  of  variance  model 
allows  one  to  assess  the  relative  strength  (weight)  of  variance  components 
attributable  to  convergent  and  discriminant  validity,  method  or  source  bias 
(halo),  and  error.  Table  1A  presents  the  results  of  variance  component  and 
indexed  calculations  for  the  LBDQ-XII  multirater  matrix.  It  can  be  seen  that 
a  modest  amount  of  the  indexed  variance  (.20)  can  be  attributed  to  rater 
source  bias  (i.e.,  'halo"  in  the  rating  situation).  An  approximately  equal 
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proportion  of  the  variance  (.19)  can  be  attributed  to  convergent  validity 
among  multiple  rater  sources  using  the  LBDQ-XII  instrument.  Finally,  almost 
none  (.01)  of  the  variance  is  attributable  to  discriminant  validity  (i.e.,  the 
subscales  of  the  LBDQ-XII  are  highly  intercorrelated).  In  summary,  this 
statistical  analysis  does  not  provide  strong  support  for  either  convergent  nor 
discriminant  validity  (when  compared  to  halo  and  error  terms)  among  multiple 
rater  sources  using  the  LBDQ-XII  instrument. 

Insert  Table  1  About  Here 


MTMM  Analysis  of  the  MBS 

The  correlation  matrix  for  the  MBS  questionnaire  showed  a  pattern  of 
results  quite  similar  to  those  produced  for  the  LBDQ-XII  instrument.  The 
results  Indicated  that  different  rater  sources  do  not  tend  to  agree  highly 
enough  (on  the  validity  diagonals)  to  argue  that  convergent  validity  existed 
to  any  great  extent.  Also  similar  to  the  LBDQ-XII,  the  results  for  the  MBS 
did  not  appear  to  satisfy  the  requirements  for  discriminant  validity  among  the 
behavioral  categories. 

The  results  of  the  statistical  analysis  for  the  MBS  shown  in  Table  IB 
indicates  that  a  moderate  amount  of  the  indexed  variance  (.29)  can  be  attribu¬ 
ted  to  rater  source  bias  (halo).  Less  of  the  variance  is  attributable  to 
convergent  validity  (.21),  and  almost  none  (.01)  to  discriminant  validity.  In 
summary,  these  results  do  not  provide  strong  support  for  either  convergent  or 
discriminant  val^lty  (when  compared  to  halo  and  error  terms)  among  multiple 
rater  sources  using  the  MBS  instrument. 
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MTMM  Analysis  of  the  LOS 

The  correlation  matrix  for  the  MTMM  analysis  of  the  LOS  are  presented  in 
Table  2.  Correlation  coefficients  (calculations  used  the  Spearman  Rho  because 
of  the  nominal  data)  between  participant  and  outside  observers  tend  to  agree 
substantially  enough  (see  the  validity  diagonal)  to  argue  that  convergent 
validity  exists  to  a  moderate  degree,  with  the  possible  exception  of  the 
infrequently  occurring  "Managing  Conflict"  behavioral  category.  With  respect 
to  discriminant  validity  criteria,  the  results  are  also  moderately  positive. 

That  is,  the  pattern  of  correlation  coefficients  in  the  validity  diagonal  is 

t 

generally  gre  '.ter  than  the  correlation  coefficients  found  in  the  two  adjacent  I 

i 

heterobehavior  blocks  (dotted  triangles),  with  the  exception  of  "Managing 
Conflict."  The  pattern  of  correlation  coefficients  in  the  validity  diagonal 
is  also  greater  than  the  correlation  coefficients  found  in  the  monomethod- 
heterobehavior  blocks  (solid  triangles).  Finally,  the  same  pattern  of  be¬ 
havior  intercorrelations  are  found  in  all  of  the  heterobehavior  triangles  of 
both  the  monomethod  and  heteromethod  blocks,  even  though  there  are  some 
differences  in  the  general  level  of  correlations  involved. 

Insert  Table  2  About  Here 

Results  for  the  statistical  analysis  of  the  LOS  matrix  are  provided  in 
Table  1C.  It  can  be  seen  from  this  table  that  a  substantial  proportion  of  the 
indexed  variance  is  attributable  to  discriminant  validity  (.68),  somewhat  less 
to  convergent  validity  (.42),  and  almost  none  (.01)  to  halo  bias.  In  total, 
the  evidence  for  convergent  and  discriminant  validity  appears  to  be  relatively 
strong  for  the  LOS  when  multiple  rater  sources  are  considered  measurement  by 
more  than  one  method.  Once  again,  however,  a  word  of  caution  is  in  order. 

One  could  question  whether  the  participant  and  outside  observers  really  repre- 


sent  multiple  methods  in  a  validity  analysis  of  the  LOS.  They  are  certainly 
different  sources  and  different  sources  have  traditionally  been  used  as 
different  methods,  but  there  are  obviously  some  problems  with  making  this 
strictly  a  validity  test.  On  the  other  hand,  validity  assessment  is  judgemen¬ 
tal  and  this  analysis  is  beginning  input  into  such  a  judgemental  process.  In 
addition,  however,  this  MTMM  analysis  of  the  LOS  does  provide  empirical  input 
for  an  alternative  assessment  to  the  interrater  evaluation  of  reliability  of 
the  LOS. 

The  LOS  and  Questionnaires  as  Multiple  Methods 

To  get  around  the  potential  problems  associated  with  the  use  of  multiple 
sources  as  multiple  methods,  an  attempt  was  made  to  compare  similar  categories 
of  the  LOS  (observation  method)  and  LBDQ-XII  and  MBS  (questionnaire 
methods).  This,  of  course,  represents  a  classic  multiple  methods  analysis. 
However,  because  there  are  widely  different  behavioral  categories  for  each 
insrument,  a  way  of  collapsing  categories  for  comparison  purposes  was  needed. 

The  data  for  the  LBDQ-XII  and  MBS  were  first  submitted  to  a  principal 
components  factor  analysis  with  varimax  rotation  in  order  to  reduce  the  indi¬ 
vidual  categories  to  factors  which  could  be  reasonably  compared.  The  factor 
analysis  for  the  LBDQ-XII  resulted  in  a  two  factor  solution  (Consideration  and 
Initiation  of  Structure),  accounting  for  64  percent  of  the  cumulative  vari¬ 
ance.  The  factor  analysis  for  the  MBS  resulted  in  a  three  factor  solution 
(Consideration,  Initiation  of  Structure,  and  Conflict  Management),  accounting 
for  58  percent  of  the  variance.  However,  since  the  third  factor  (Conflict 
Management)  contributed  only  .06  percent  to  the  cumulative  proportion  of 
explained  variance,  the  data  were  subsequently  forced  into  a  two  factor  solu¬ 
tion  (Consideration  and  Initiation  of  Structure)  for  comparison  purposes.  A 
factor  analysis  for  the  LOS  was  not  possible  because  the  observers  were  in- 


structed  to  check  as  many  behaviors  as  occurred  within  a  given  ten  minute  time 
period.  Since  more  than  one  behavior  was  often  observed  during  thistime 
frame,  there  would  be  high  intercorrelations  among  LOS  categories  and  thus 
trying  to  determine  a  factor  structure  would  be  meaningless.  As  a  result,  for 
comparison  purposes  the  LOS  categories  were  forced  on  a  conceptual  basis  into 
Consideration  and  Initiation  of  Structure  factors  like  to  the  results  from  the 
LBDQ-XII  and  MBS  factor  analyses.  Figure  2  shows  how  the  behavioral  categor¬ 
ies  for  each  instrument  were  collapsed  into  Consideration  and  Initiation  of 
Structure . 

Insert  Figure  2  About  Here 

The  MTMM  correlation  matrix  for  this  multiple  methods  analysis  is 
presented  in  Table  3.  Several  different  combinations  of  behavioral  categories 
collapsed  into  Consideration  and  Initiation  of  Structure  were  also  analyzed 
with  results  quite  similar  to  those  found  in  Table  3.  It  can  be  seen  from 
that  with  the  possible  exception  of  the  modest  degree  of  convergence  between 
the  LOS  and  MBS  Consideration  factor,  there  is  little  evidence  of  convergence 
in  the  validity  diagonals  between  the  LOS  observational  method  and  either  of 
the  LBDQ-XII  or  MBS  questionnaire  methods.  There  is,  however,  a  substantial 
amount  of  convergence  in  the  validity  diagonal  between  the  LBDQ-XII  and  MBS — 
both  of  which  are  questionnaire-based  methods.  None  of  the  criteria  for 
discriminant  validity  were  satisfied  to  any  great  extent. 

Insert  Table  3  About  Here 

The  results  of  the  statistical  analysis  of  the  MTMM  matrix  is  shown  in 
Table  ID.  It  can  be  seen  that  a  substantial  proportion  of  the  indexed  vari¬ 
ance  (.54)  is  attributable  to  convergent  validity  (for  the  most  part  between 


22 


Che  two  questionnaire  methods).  A  substantial  proportion  (.55)  of  the  vari¬ 
ance,  however,  is  also  attributed  to  halo  bias  in  the  rating  situation  (i.e., 
high  scale  intercorrelations  between  C  and  IS  for  each  instrument).  Finally, 
there  was  only  a  very  modest  amount  of  variance  contributed  by  discriminant 
validity  (.10).  In  summary,  these  results  provide  only  minor  support  for  the 
construct  validity  of  any  of  the  three  leadership  behavior  measures  analyzed 
by  the  MTMM. 

To  make  a  more  directly  comparable  multiple  methods  analysis,  the  LOS  and 
the  self-report  percentage  of  time  usage  questionnaire  containing  the  same 
behavioral  categories  were  used  as  two  different  methods.  Table  4  shows  that 
in  the  validity  diagonal  there  was  significant  convergence  between  methods 
(the  LOS  and  the  self-report  time  usage  questionnaire)  for  six  of  the  twelve 
categories  with  three  more  approaching  (.17)  significance.  There  was,  how¬ 
ever,  little  evidence  of  convergence  for  the  categories  of  "Decision 
Making/Problem  Solving,"  "Disciplining/Punishing,"  and  "Exchanging  Routine 
Information."  It  is  also  interesting  to  note  that  there  was  greater  conver¬ 
gence  between  the  self  estimate  of  time  usage  and  participant  observers  (who 
in  nearly  all  Instances  were  the  subordinates  of  target  managers)  than  in 
self-subordinate  comparisons  for  either  the  LBDQ-XII  or  MBS  questionnaire 
measures.  None  of  the  criteria  for  discriminant  validity  seemed  to  be  satis¬ 
fied  to  any  great  extent. 

Insert  Table  4  About  Here 

The  statistical  results  shown  in  Table  IE  indicates  that  a  moderate 
amount  of  the  indexed  variance  was  attributable  to  both  convergent  validity 
(.31)  and  halo  bias  (.33),  with  the  remaining  proportion  (.15)  due  to  discrim¬ 
inant  validity.  In  summary,  these  results  provide  moderate  support  for 
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convergent  validity,  but  less  for  the  discriminant  validity  of  LOS  when  the 
multiple  methods  are  considered  to  be  the  observation  system  (LOS)  and  a 
directly  comparable  self  estimate  of  time  usage  questionnaire. 

Discussion 

This  study  provides  data  to  begin  to  analyze  the  realiability  and  valid¬ 
ity  of  a  newly  developing  observation  system  to  measure  leader  behavior  in 
natural  settings.  Simple  interrater  agreement  reliability  was  found  to  be 
quite  high,  but  more  analysis  was  provided.  In  particular,  the  multitrait- 
multimethod  (MTMM)  technique  was  used.  Although  there  are  certain  limitations 
with  the  MTMM  approach,  (e.g.  Kalleborg  and  Klugel  (1975)  have  shown  that  the 
building  block  correlations  of  the  MTMM  matrix  are  all  complexly  determined, 
and,  therefore,  the  comparisons  involved  in  the  MTMM  criteria  will  also  be 
complexly  determined]  it  was  deemed  to  provide  the  most  appropriate  and 

comprehensive  analysis  of  the  LOS  as  is  possible  at  this  point  of  its  develop-  ] 

raent . 

In  general,  the  results  of  the  MTMM  analysis  indicates  that  when  multiple 
rater  sources  (self,  superior,  peer,  and  subordinte)  were  considered  measure¬ 
ment  by  more  than  one  method  (as  widely  cited  MTMM  analyses  have  done  in  the 
past,  e.g.,  Campbell  &  Fiske,  1959  and  Lawler,  1967),  the  questionnaire-based 
LBDQ-XII  and  MBS  measures  were  not  demonstrated  to  have  support  for  either 
convergent  or  discriminant  validity.  Instead,  a  rather  substantial  within- 
rater  source  halo  bias  existed.  Raters  using  these  questionnaire  methods 
generally  failed  to  discriminate  among  presumably  independent  categories. 

That  is,  a  given  source  tended  to  rate  the  leader  on  all  behavioral  categories 
in  much  the  same  manner  (as  evidenced  by  high  scale  intercorrelations  within 
rater  source).  In  addition,  chere  was  little  agreement  (convergence)  in  the 
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ratings  of  the  leader  among  different  rater  sources.  Each  rater  source  ap¬ 
parently  perceived  the  leader’s  behavior  quite  differently.  This  finding  has 
important  implications  for  the  evaluation  of  the  validity  of  these  question¬ 
naires  and  for  analyzing  the  situational  determinants  of  leader  behavior. 

With  respect  to  the  LOS,  results  of  the  KTMM  analysis  indicated  at  least 
moderate  support  for  both  convergent  and  discriminant  validity  when  partici¬ 
pant  observers  (generally  subordinates)  and  outside  observers  were  considered 
measurement  by  more  than  one  method.  In  other  words,  participant  and  outside 
observers  tended  to  agree  when  describing  the  behavior  of  a  leader  and  tended 
to  discriminate  among  behavioral  categories. 

Compared  to  the  questionnaire  methods,  the  LOS  faired  better  in  this 
analysis  using  multiple  sources  as  multiple  methods.  Although  the  rater 
sources  were  somewhat  different  and  more  comprehensive  for  the  questionnaire 
measures,  the  comparison  shows  that  the  LOS  indices  for  both  convergent  and 
discriminant  validities  are  greater  and  the  halo  problem  less.  This  result, 
of  course,  is  not  convincing  evidence,  nor  is  there  any  intent  to  prove,  that 
the  observation  system  is  superior  to  the  questionnaire  methods.  It  must  be 
remembered  that  the  LOS  data  are  based  on  a  fixed  ten  minute  observation 
period  that  Is  common  to  both  rater  sources.  The  LBDQ  and  MBS,  on  the  other 
hand,  are  based  on  an  open  ended  and  unspecified  period  of  time.  Another 
problem  In  making  direct  comparisons  is  that  both  rater  sources  in  the  LOS  are 
watching  the  same  target  leader  at  the  same  time  while  the  LBDQ  and  MBS  draw 
from  pooled  data  representing  the  ave  ■  .eadership  style  (ALS)  of  the  target 
leaders.  Despite  these  and  other  potc  itial  problems  with  making  direct 
comparisons  in  this  study,  the  results  can  serve  as  a  stimulus  and  point  of 
departure  to  examine  and  interpret  some  of  the  problems  that  questionnaires 
may  have  relative  to  observational  methods  and  help  justify  the  effort  for 
developing  an  observational  system  to  supplement  questionnaire  measures. 
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For  example,  one  Interpretation  from  these  comparative  results  would  be 
that  the  leadership  questionnaire  measures  may  be  susceptible  to  high  degrees 
of  selective  recall  and  halo  bias  on  the  part  of  the  raters.  On  the  other 
hand,  an  observational  system  such  as  the  LOS  may  be  less  susceptible  to 
selective  recall  (i.e.,  the  lag  between  observation  and  recorded  benavior  is 
more  immediate  than  in  questionnaires).  In  addition,  halo  bias  may  be  mini¬ 
mized  when  well-trained  participant  and  outside  observers  are  used  to  gather 
leader  behavior  data.  For  these  reasons  alone,  continued  efforts  toward  the 
development  and  use  of  observational  methods  for  measuring  leader  behavior 
seem  justified. 

The  results  from  this  study  would  also  seem  to  indicate  that  any  one 
rater  source  using  questionnaire  measures  would  not  provide  an  adequate 
assessment  of  leader  behavior.  The  typical  practice  of  correlating  subordi¬ 
nate  questionnaire  measures  of  leader  behavior  with  performance  ratings  of 
leaders  based  on  perceptions  by  superiors  may  be  like  comparing  apples  and 
oranges.  This  study  clearly  indicates  that  various  rater  sources  filling  out 
a  questionnaire  perceive  the  behavior  of  a  leader  quite  differently.  An 
"appropriate”  rater  source  would  seem  to  depend  more  on  the  criterion  measure 
of  interest.  For  example,  if  the  criterion  measure  of  interest  is  performance 
ratings  by  superiors,  then  perhaps  the  appropriate  rater  source  to  describe 
the  ratee's  behavior  is  the  superior.  If  subordinate  satisfaction  is  the 
criterion  measure  of  interest,  then  perhaps  the  appropriate  rater  source  is 
subordinate  perceptions  of  leader  behavior,  and  so  on.  This  approach,  how¬ 
ever,  would  also  run  the  risk  that  any  significant  results  may  be  due  to 
common  source  (rater)  variance  unless  there  was  a  considerable  time  lag  be¬ 
tween  the  administration  of  the  measures. 


When  the  MTMM  analysis  used  the  LOS  (observations)  and  MBS  and  LBDQ 
(questionnaires)  as  multiple  methods,  the  results  were  less  encouraging  for 
the  LOS  than  when  multiple  rater  sources  were  used  as  multiple  methods.  When 
Consideration  and  Initiation  of  Structure  types  of  factors  for  the  LOS,  LBDQ- 
XII,  and  MBS  instruments  were  compared,  results  indicated:  (1)  a  modest 
degree  of  convergence  between  the  LOS  and  MBS  Consideration  factor,  but  not 
for  Initiation  of  Structure;  (2)  no  significant  convergence  between  the  LOS 
and  LBDQ-XII  for  either  factor;  and  (3)  a  substantial  degree  of  convergence 
between  the  MBS  and  LBDQ-XII  for  both  factors.  The  latter  result  could  be 
attributed  to  a  methodological  artifact  (i.e.,  both  MBS  and  LBDQ-XII  are 
questionnaire  methods).  The  evidence  to  support  the  discriminant  validity  of 
any  of  these  instruments  was  very  slight  in  this  portion  of  the  analysis.  It 
must  be  remembered,  however,  that  especially  in  the  case  of  the  LOS,  there 
were  not  directly  comparable  behavioral  categories  with  the  other  methods. 
When  an  analysis  was  made  with  two  different  methods  (the  LOS  and  the  Self 
Estimate  of  Time  Use  Questionnaire)  containing  the  same  behavioral  categories, 
then  the  LOS  did  faire  better.  A  moderate  degree  of  convergent  validity  was 
evidenced,  but  less  support  was  shown  for  discriminant  validity. 


Conclus ions 


It  should  be  emphasized  once  again  that  this  study  only  provide  a  beginn¬ 
ing  for  analyzing  a  newly  developing  observation  system  of  measuring  leader 
behavior  in  natural  settings.  There  is  no  intention  that  the  results  from 
this  study  can  be  construed  as  conclusive.  Rather,  the  importance  of  this 
study  lies  in  demonstrating  the  potential  reliability  and  validity  of  a 
supplemental  method  to  questionnaires  for  measuring  leader  behavior  in  natural 
settings.  The  high  interrater  agreement  percentage  contributes  to  the  evalua¬ 
tion  of  the  reliability  of  the  LOS  approach  and  the  MTMM  analysis,  especially 
when  multiple  rater  sources  are  considered  to  be  multiple  methods,  contributes 
to  its  reliability  and  validity  evaluation.  The  MTMM  analysis  using  observat¬ 
ion  and  questionnaires  as  multiple  methods  was  not  as  encouraging.  However, 
without  directly  comparable  behavioral  categories  across  the  methods,  these 
results  may  not  be  surprising.  When  the  questionnaire  method  did  use  directly 
comparable  categories  (the  Self  Estimate  of  Time  Usage),  the  results  of  the 
validity  analysis  were  more  encouraging. 

An  obvious  need  for  future  study  would  be  to  make  a  comparison  between 
questionnaire  and  observation  methods  that  have  directly  comparable  categor¬ 
ies.  If  there  is  demonstrated  support  for  the  validity  of  the  widely  used 
LBDQ  or  the  new  MBS,  then  the  more  practical  and  easy  to  use  questionnaire 
method  should  be  used  as  an  important,  but  not  only ,  data  gathering  technique 
for  leadership  research  and  application  techniques.  The  same  is  true  of  the 
observation  system.  By  using  both  questionnaires  and  observational  tech¬ 
niques,  a  network  of  concordance  among  multiple  methods  of  measurement  can 
result.  Such  a  multiple  methods  approach  seems  to  be  the  most  feasible  way  of 
obtaining  a  reliable  and  valid  measure  of  extremely  complex  leader  behavior. 

As  Nunnally  points  out:  "validity  usually  is  a  matter  of  degree  rather  than  an 
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all-or- none  property,  and  validation  is  an  unending  process.  .  .  New  evidence 
may  suggest  modifications  of  an  existing  measure  or  the  development  of  a  new 
and  better  approach  to  measuring  the  attribute  in  question"  (1978,  p.  87). 

This  study  represents  but  one  step  in  this  continuing  effort  to  obtain  reli¬ 
able  and  valid  measures  of  leader  behavior. 
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Figure  1 

The  LOS  Categories  and  Behavioral  Descriptors 


1.  PLANN I NG/COORO I  MAT  I NG 

a.  setting  goals  &  objectives 

b.  defining  tasks  needed  to 
seconp I  I sh  goals 

c.  scheduling  employees,  1 1  satab  I  es 

d.  assigning  tasks  and  providing 
routine  Instructions 

e  .  coordinating  activities  of 
different  subs  to  keep  aork 
running  smoothly 

f,  organizing  the  aork 

2.  STAFFING 

a.  developing  Job  descriptions  for 
position  openings 

b.  review  applications 

c.  Interview  applicants 

d .  hiring 

e.  contacting  applicants  to  Inform 
them  of  being  hired  or  not 

t.  'filling  In*  where  needed 

3.  TRA I N I  MG/ DE  y ELOPING 

a.  orienting  employees,  arranging 
tor  training  seminars,  etc. 

b.  clarifying  roles,  duties.  Job 
descriptions 

c.  coaching,  mentoring,  walking 
subordinates  through  task 

d.  helping  subordinate  with  personal 
development  plans 

4.  DECISION  NAK 1 NG/PROBLEN  SOLVING 

a.  defining  problems 

b.  choosing  between  2  or  more 
alternatives  or  strategies 

c.  handling  day-to-day  operational 


7.  NON  I  TOR  I NG/CONTROLL I N6  PERFORNANCE 

a.  Inspecting  work 

b.  walking  around  and  checking 
things  out,  touring 

c.  monitoring  performance  data 
(e.g.,  computer  printouts, 
production,  financial  reports) 

d.  preventative  maintenance 

8.  NOT  I  VAT  I NG/RE I NFORC I NG 

a.  allocating  formal  organizational 
rewards 

b.  asking  for  Input,  participation 

c.  conveying  appreciation, 
compl Iments 

d.  g I v Inc  credit  where  due 

e.  listening  to  suggestions 

t.  giving  positive  performance 
feedback 

g.  Increasing  Job  challenge 

h.  delegating  responsibility  4 
authority 

I.  letting  subordinates  determine 
how  to  do  their  own  work 

J.  sticking  up  for  the  group  to 
superiors  and  others,  backing  a 
subordinate 

9.  DISCIPLINING/PUNISHING 

a.  enforcing  rules  and  policies 

b.  non  verbal  glaring,  harassment 

C.  demotion,  firing,  layoff 

d.  any  formal  organizational 
reprimand  or  notice 

e.  "chewing  out*  a  subordinate, 
criticizing 

f.  giving  negative  performance 


crises  as  they  arise 

d.  weigh  the  trade-offs;  cost 
benefit  analyses 

e.  actually  deciding  what,  to  do 

t,  developing  new  procedures  to 

Increase  efficiency 

PROCESSING  PAPERNORK 

a.  processing  mall 

b.  reading  reports.  In-box 

c.  writing  reports,  memos,  letters, 
etc  . 

d.  routine  financial  reporting  nnd 
book  keep  I ng 

e.  general  desk  work 

EXCHANGING  ROUTINE  I NFORNAT I  ON 

a.  answering  routine  procedural 
guest  I ons 

b.  receiving  and  disseminating 
reauested  Information 

e.  conveying  results  of  meetings 

d.  giving  or  receiving  routine 
Information  over  the  phone 

a.  staff  meetings  of  an 

Informational  nature  (e.g., 
status  updates,  new  company 
policies,  etc .  ) 


feedback 

10.  INTERACTING  WITH  OUTSIDERS 

a.  public  relations 

b.  customers 

c.  contacts  with  suppliers,  vendors 

d.  external  meetings 

a.  community  service  activities 

It,  NANAGING  CONFLICT 

a.  managing  Interpersonal  conflict 
between  subordinates  or  others 

b.  appealing  to  higher  authority  to 
resolve  a  dispute 

c.  appealing  to  3rd  party 
negotl ators 

d.  trying  to  get  cooperation  or 
consensus  between  conflicting 
parties 

a.  attempting  to  resolve  conflicts 
between  subordinate  end  self 

12.  SOCIALIZl NG/POL ITICKIN6 

a.  nonwork  related  chit  chat  (e.g., 
family  or  personal  matters) 

b.  Informal  'Joking  around,'  B.S. 

c.  discussing  rumors,  hearsay, 
grapav I  no 

d.  complaining,  griping,  putting 
others  down 

e.  politicking,  gamesmanship 


Figure  2 

Collapsing  of  Behavioral  Categories  for  Each  Method  into 
Consideration  and  Initiation  of  Structure 


Initiation  of  Structure 

Coordinating 
Planning 
Problem  Solving 
Role  Clarification 
Goal  Setting 
Training 


Initiation  of  Structure 

Integration 
Production  Emphasis 
Representation 
Superior  Orientation 
Initiation  of  Structure 


Initiation  of  Structure 

Planning/Coordinating 

Decision-Making/Problem 

Solving 

Monitorial/Controlling 

Performance 

Staffing 

Exchanging  Routine 
Information 
Training 


MBS 

Consideration 

Autonomy /Delegation 
Consideration 

Facilitating  Subordinate  Work 
Criticism/Discipline 
Conflict  Management 
Facilitating  Group  Interaction 

LBDQ-XII 

Consideration 

Consideration 
Tolerance  of  Freedom 
Tolerance  of  Uncertainty 


LOS 


Consideration 


Motivating/Reinforcing 
Disciplining/Punishing 
Managing  Conflict 
Socializing/Politicking 


Table  1A 

Components  and  Indices  for  the  LBDQ-XII  MTMM  Using  Multiraters 
(Self,  Superior,  Subordinate  and  Peer) 


Source* 

Variance  Components 

Indices** 

L  (convergent  validity) 

.168 

.19 

L  x  B  (discriminant  validity) 

.009 

.01 

L  x  S  (halo) 

.176 

.20 

Error 

.708 

*L  «  Target  Leader  **Variance  components  standardized 

B  *  Behavior 

S  »  Source  or  Rater 

error  term 

Table  IB 

Components  and  Indices  for  the  MBS  MTMM  Using  Multiraters 
(Self,  Superior,  Subordinate  and  Peer) 

Source 

Variance  Components 

Indices 

L 

(convergent  validity)  .167 

.21 

L  x  B 

(discriminant  validity)  .009 

.01 

L  x  S 
Error 

(halo)  .256 

.632 

.29 

Components  and  Indices 
(Participant 

Table  1C 

for  the  LOS  MTMM  Using 
and  Outside  Observers) 

Multiraters 

Source 

Variance  Components 

Indices 

L  (convergent  validity) 

.196 

.42 

L  x  B  (discriminant  validity) 

.584 

.68 

L  x  S  (halo) 

.003 

.01 

Error 

.268 

Components  ar  ! 

of 

Table  ID 

Indices  for  the  Multiple  Methods 
LBDQ-XII,  MBS,  and  LOS 

Source 

Variance  Components 

Indices 

L 

(convergent  validity) 

.381 

.54 

L  x  B 

(discriminant  validity) 

.038 

.10 

L  x  S 

(halo) 

.407 

.55 

Error 

.328 

Components  and  Indices 
(Observations 

Table  IE 

for  the  LOS  MTMM  Using  Multiple  Methods 
and  Self  Estimate  Questionnaire) 

Source 

Variance  Components 

Indices 

L  (convergent  validity) 

.239 

.31 

L  x  B  (discriminant  validity) 

.097 

.15 

L  x  S  (halo) 

.265 

.33 

Error 

.540 

Table  2 

LOS  Multirater  Matrix 
(Participant  and  Outside  Observers) 


The  figures  enclosed  in  parentheses  are  the  validity  diagonals;  the  figures 
enclosed  within  solid  triangles  are  the  monoraethod-heterobehavior  blocks;  and 
the  figures  enclosed  within  dotted  triangles  are  the  heteromethod- 
heterobehavior  blocks.) 
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Table  3 

Multimethod  Matrix  for  LBDQ,  MBS,  and  LOS 
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Table  A 

Multiraethod  Matrix: 

LOS  and  Self  Estimate  of  Time  Usage  Questionnaire 


(The  figures  enclosed  in  parentheses  are  the  validity  diagonals;  the  figures 
enclosed  within  solid  triangles  are  the  nomomethod-heterobehavior  blocks;  and  the 
figures  enclosed  within  dotted  triangles  are  the  heteromethod-heterobehavior 
blocks . ) 
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Department  of  Administration  Sciences 
Monterey,  CA  93940 

Superintendent 

ATTN:  Director  of  Research 

Naval  Academy,  U.S. 

Annapolis,  MD  21402 
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LIST  7 
HRM 


Officer  In  Charge 

Human  Resource  Management  Detachment 
Naval  Air  Station 
Alameda,  CA  94591 

Officer  In  Charge 

Human  Resource  Management  Detachment 
Naval  Submarine  Base  New  London 
P.0.  Box  81 
Groton,  CT  06340 

Officer  In  Charge 

Human  Resource  Management  Division 
Naval  Air  Station 
May port,  FL  32228 

Commanding  Officer 

Human  Resource  Management  Center 

Pearl  Harbor,  HI  96860 

Commander  in  Chief 

Human  Resource  Management  Division 

U.S.  Pacific  Fleet 

Pearl  Harbor,  HI  96860 

Officer  In  Charge 

Human  Resource  Management  Detachment 
Naval  Base 

Charleston,  SC  29408 

Commanding  Officer 
Human  Resource  Management  School 
Naval  Air  Station  Memphis 
Millington,  TN  38054 

Human  Resource  Management  School 
Naval  Air  Station  Memphis  (96) 
Millington,  TN  38054 
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List  7  (Continued) 


Commanding  Officer 
Human  Resource  Management  Center 
1300  Wilson  Boulevard 
Arlington,  VA  22209 

Commanding  Officer 
Human  Resource  Management  Center 
5621-23  Tidewater  Drive 
Norfolk,  VA  23511 

Commander  in  Chief 

Human  Resource  Management  Division 

U.S.  Atlantic  Fleet 

Norfolk,  VA  23511 

Officer  Ip  Charge 

Human  Resource  Management  Detachment 
Naval  Air  Station  Whidbey  Island 
Oak  Harbor,  WA  98278 

Commanding  Officer 

Human  Resource  Management  Center 

Box  23 

PPO  New  York  09510 

Commander  in  Chief 

Human  Resource  Management  Division 

U.S.  Naval  Force  Europe 

PPO  New  York  09510 

Officer  in  Charge 

Human  Resource  Management  Detachment 

Box  60 

PPO  San  Francisco  96651 
Officer  in  Charge 

Hunan  Resource  Management  Detachment 

C0MNAVF0RJAPAN 

PPO  Seattle  98762 
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LIST  8 

NAVY  MISCELLANEOUS 


Naval  Military  Personnel  Command  (2  copies) 
HRM  Department  (NMPC-6) 

Washington,  DC  20350 


Naval  Training  Analysis 
and  Evaluation  Croup 
Orlando,  FL  32813 

Commanding  Officer 
ATTN:  TIC,  Bldg.  2068 
Naval  Training  Equipment  Center 
Orlando,  FL  32S13 

Chief  of  Naval  Education 
and  Training  (N-5) 

Director,  Research  Development, 
Test  and  Evaluation 
Naval  Air  Station 
Pensacola,  FL  32508 

Chief  of  Naval  Technical  Training 
ATTN:  Dr.  Norman  Kerr,  Code  017 
NAS  Memphis  (75) 

Millington,  TN  38054 

Navy  Recruiting  Command 

Head,  Research  and  Analysis  Branch 

Code  434 ,  Room  8001 

801  North  Randolph  Street 

Arlington,  VA  22203 

Commanding  Officer 
OSS  Carl  Vinson  (CVN-70) 

Newport  News  Shipbuilding  & 

Drydock  Company 
Newport  News,  VA  23607 


Headquarters,  U.S.  Marine  Corps 
Code  MPI-20 
Washington,  DC  20380 

Headquarters,  U.S.  Marine  Corps 
ATTN:  Dr.  A.  L.  Slafkosky, 

Code  RD-1 

Washington,  DC  20380 

Education  Advisor 
Education  Center  (E031) 

MCDEC 

Quantico,  VA  22134 

Commanding  Officer 
Education  Center  (E031) 

MCDEC 

Quantico,  VA  22134 

Commanding  Officer 
U.S.  Marine  Corps 
Command  and  Staff  College 
Quantico,  VA  22134 


LIST  9 
USMC 
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LIST  13 
AIR  FORCE 


LIST  12 
ARMY 


Air  University  Library/LSE  76-443 
Maxwell  AFB,  AL  36112 

COL  John  W.  Williams,  Jr. 

Head,  Department  of  Behavioral 

Science  and  Leadership 

U.S.  Air  Force  Academy,  CO  80840 

MAJ  Robert  Gregory 
UsAFA/DFBL 

U.S.  Air  Force  Academy,  CO  80840 

AFOSR/NL  (Dr.  Fregly) 

Building  410 
Bolling  AFB 
Washington,  DC  20332 

LTCOL  Don  L.  Presar 
Department  of  the  Air  Force 
AF/MPXHM 
Pentagon 

Washington,  DC  20330 

Technical  Director 
AFHRL/MO(T) 

Brooks  AFB 

San  Antonio,  TX  78235 

AFMPC/MPCYPR 
Randolph  AFB,  TX  78150 


Headquarters,  F0RSC0M 

ATTN :  AFPR-HR 

Ft.  McPherson,  GA  30330 

Army  Research  Institute 
Field  Unit  -  Leavenworth 
P.0.  Box  3122 

Fort  Leavenworth,  KS  66027 

Technical  Director 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Director 

Systems  Research  Laboratory 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Director 

Army  Research  Institute 
Training  Research  Laboratory 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Dr.  T.  0.  Jacobs 
Code  PERI-IM 
Army  Research  Institute 
5001  Elsenhower  Avenue 
Alexandria,  VA  22333 

COL  Howard  Prince 

Head,  Department  of  Behavior 

Science  and  Leadership 

U.S.  Military  Academy,  New  York  10996 


